Method and device for displaying details of a texture of a three-dimensional object

ABSTRACT

A computer implemented method for displaying details of a texture of a three-dimensional, 3D, object, wherein the texture comprises a periodic pattern, is provided. The method comprises, while zooming in on the 3D object: determining a portion of the 3D object, wherein the determined portion of the 3D object corresponds to a zoom level; displaying the determined portion of the 3D object including a corresponding portion of the texture. The method further comprises, upon the zoom level reaching a zoom-in threshold, executing a 3D to 2D transition comprising: identifying a two-dimensional, 2D, image representing the texture of the currently displayed portion of the 3D object from a set of 2D images, each 2D image of the set of 2D images representing the texture of the 3D object at a specific of portion of the 3D object, wherein at least one 2D image of the set of 2D images represents the texture of the 3D object at a plurality of portions of the 3D object; wherein a resolution of the 2D image is higher than a resolution of the texture of the portion of the 3D object at the zoom-in threshold.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims priority to European Patent Application No.20215009.0, filed Dec. 17, 2020, the disclosure of which is herebyincorporated by reference in its entirety. To the extent appropriate, aclaim of priority is made to the above-disclosed application.

FIELD OF TECHNOLOGY

The present disclosure belongs to methods and devices for displayingdetails of a texture of a three-dimensional object.

BACKGROUND

Rendering and displaying three-dimensional, 3D, objects on, e.g.,websites or in software applications for online commerce has recentlygrown immensely popular among a wide spectrum of companies. Thanks toimproved internet-connection bandwidth, and/or improved processing powerof electronic devices, a user may see, rotate, or perform othermanipulations of a rendered 3D object, such as changing the color or theclothing of the 3D object.

A rendered 3D object, captured by, e.g., a 3D scanner, may provide anadequate visual experience to a user in view of an overall impressionsuch as the shape or the color of the 3D object. However, the resolutionof a texture of such a rendered 3D object is often relatively scarce dueto the large amount of data that otherwise needs to be transferred fordense point clouds associated with such a high resolution 3D object.Moreover, despite being capable of providing a dense point cloud of a 3Dobject, 3D scanners are often incapable to capture subtle details of thesurface of the 3D object, such as a surface texture and/or a detailedpattern of a clothing of a piece of furniture, due to the notoriousdifficulty of measuring short distances by such devices.

Hence, there is a need for an approach to provide deep zooming on a 3Dobject for enabling rendering of subtle details of the texture of the 3Dobject.

SUMMARY

Thus, it is an object of the invention to provide a method to providedeep zooming on a 3D object for enabling rendering of subtle details ofthe texture of the 3D object.

According to a first aspect, there is provided a computer implementedmethod for displaying details of a texture of a three-dimensional, 3D,object, the texture comprising a periodic pattern, the methodcomprising:

-   -   while zooming in on the 3D object:        -   determining a portion of the 3D object, wherein the            determined portion of the 3D object corresponds to a zoom            level, and        -   displaying the determined portion of the 3D object including            a corresponding portion of the texture, and    -   upon the zoom level reaching a zoom-in threshold, executing a 3D        to 2D transition comprising:        -   identifying a two-dimensional, 2D, image representing the            texture of the currently displayed portion of the 3D object            from a set of 2D images, each 2D image of the set of 2D            images representing the texture of the 3D object at a            specific of portion of the 3D object, wherein at least one            2D image of the set of 2D images represents the texture of            the 3D object at a plurality of portions of the 3D object,            and        -   displaying the 2D image, wherein a resolution of the 2D            image is higher than a resolution of the texture of the            portion of the 3D object at the zoom-in threshold.

Within the present specification, a “texture” refers to a digitalrepresentation of a surface of an object. In contrast totwo-dimensional, 2D, properties, such as color or brightness, a textureis also associated with three-dimensional, 3D, properties, such as atransparency or reflectance of the surface of the object.

The 3D object is comprised in a 3D model. Thus, the zooming on the 3Dobject, as referred to herein, is typically performed in arepresentation of the 3D model.

The zoom level, as referred to herein, is a measure of how magnified theportion of the 3D object appears on a display. A high zoom level on the3D object corresponds to a high magnification of the texture of the 3Dobject. Hence, a high zoom level corresponds to a close-up view of thetexture of the 3D object being, visually, a small portion of the 3Dobject compared to a lower zoom level, the lower zoom level accordinglycorresponding to a larger visible portion of the 3D object. Whenreaching a zoom-in threshold, i.e., when reaching an upper limit of themagnitude of the zoom level, a 3D to 2D transition is executed. Such anupper limit may be determined according to a minimal acceptableresolution of the zoomed in portion of the 3D object. The 3D to 2Dtransition, refers to a transition of a rendering of a zoomed in portionof the 3D object and a rendering of a 2D image comprising a higherresolution. The 2D image is thus not part of the texture of the 3Dobject but part of a set of high resolution 2D images separate from the3D object.

This may allow to further zoom in on the 2D image corresponding to thezoomed-in portion of the 3D object to further explore a high-resolutionrendering of the texture of the 3D object. Consequently, the resolutionof the texture of the 3D object can be kept at a lower level, thusreducing the memory requirements for storing the 3D object. Moreover, bygoing from rendering a 3D object, which includes shading of the textureetc., to rendering a 2D image, valuable computational resources may besaved.

As of the presumed periodic pattern of the surface of the 3D object, asingle 2D image of a periodically occurring portion of the texture ofthe 3D object may be used. Hence, when zooming in on differently locatedsuch periodically occurring portions to such a degree that the zoom-inthreshold is reached, the same 2D image may be reused at every such aperiodically occurring portion. This may allow a detailed representationof the texture of the whole surface of the 3D object represented by onlya fraction of high-resolution, HR, 2D images, thereby reducing diskspace and/or enhancing the speed of the rendering. Further, in aserver-device system architecture, bandwidth may be saved due to thelimited number of 2D images to be transferred to an electronic deviceused for zooming, and due to the lower requirements of the resolution ofthe 3D object since the deep zooming functionality being facilitated bythe 2D images.

The 3D to 2D transition may be rendered during a time period of theorder of seconds. This may allow a fast and convenient user experiencefor deep zooming to explore the details of the texture of the 3D object.

The method may further comprise, while displaying the determined portionof the 3D object, shifting a view direction of the 3D object to beparallel with a surface normal of the surface of the 3D object.

The shifting of the view direction may be made during the zooming in onthe 3D object. This may allow for a smooth user experience and animproved user-machine interface. The displaying of the 2D image may bemade upon the shifting of the view direction is completed.

The method may comprise determining an identifier of the 3D object,thereby determining the set of 2D images. The identifier may correspondto a class of objects, e.g., sofas or pillows. The identifier may alsocorrespond to a specific type of object within a class of objects, suchas a specific type of sofa or a specific type of pillow. The identifiermay also identify the pattern of texture, e.g. a clothing of the object.This may enhance the flexibility of the method.

The identification of the 2D image may comprise finding a bestcorrespondence between the portion of the 3D object and the images ofthe set of 2D images.

The finding of a best correspondence between the portion of the 3Dobject and the images of the set of 2D images may comprise inputting asample of the texture of a currently displayed portion of the 3D objectin a neural network being trained to output a 2D image representing abest match texture.

A neural network algorithm may be capable of recognizing patterns andsymbols from relatively scarce resolution images. Hence, the texture ofthe displayed portion may comprise a relatively low resolution, yet tobe detailly explored by the user by the presented method. The trainingof the neural network may allow minimizing errors while finding the bestcorrespondence and thereby enhancing accuracy and speed of the method.This may further allow the 3D model of the 3D object to be relativelylow resolution without compromising on the user experience. Hence, diskspace and bandwidth may be saved. Accordingly, CPU and/or GPU time andpower may be saved.

The finding of a best correspondence between the portion of the 3Dobject and the images of the set of 2D images may comprise an imagecomparison between a sample of the texture of a currently displayedportion of the 3D object and the textures represented by the 2D imagesin the set of 2D images. The sample of the texture may refer to thewhole visible part of the texture from a specific view angle on the 3Dobject. Alternatively, the sample of the texture may refer to portion ofthe visible part of the texture. Yet another option is that the sampleof the texture may refer to a down-sampled version, e.g., a version ofthe sample of the texture having a lower resolution. Any known imagecomparison algorithm may be used for determining a similarity measurebetween a specific image and a set of images. Such an image comparisonalgorithm may include one or more of a pixel-to-pixel comparison, edgedetection/comparison, or the like.

The image comparison algorithm may comprise a transformation of thetexture of a currently displayed portion of the 3D object to be parallelwith a surface normal of the surface of the 3D object, to facilitate theimage comparison.

The finding of a best correspondence between the portion of the 3Dobject and the images of the set of 2D images may comprise accessing aconversion table between specific portions of the 3D object and the 2Dimages in the set of 2D images. Such a conversion table may enhance thespeed and efficiency of the method.

The 3D to 2D transition may further comprise displaying a merge of theportion of the 3D object and the 2D image. The 3D to 2D transition maythus appear smooth and may thereby enhance the user experience.

The merge of the portion of the 3D object and the 2D image may be set togradually change from 100% of the portion of the 3D object to 100% ofthe 2D image. Such a gradual change may occur during a time period ofthe order of seconds. The gradual change may be linear in time or zoomlevel such that the gradual change is done at a constant rate of change.

The method may further comprise zooming and/or panning in the 2D image.By enabling further zooming in the 2D image may allow the user tofurther explore details of the texture of the 3D object. Provided a userhas zoomed in further on the 2D image, the user may pan in the 2D image.This may allow the user to conveniently explore the details of thetexture of the 3D object and thus an improved user-machine interface.

The method may further comprise, upon a zoom level in the 2D imagereaches a zoom-out threshold, executing a 2D to 3D transition comprisingdisplaying a portion of the 3D object corresponding to a currentlydisplayed portion of the 2D image. The reverse zoom direction allowszooming out for overviewing the 3D object. This may be useful, shouldthe user wish to zoom in on another portion of the 3D object beinglocated on a different location of the 3D object.

Each 2D image of the set of 2D images may represent the texture of the3D object at a plurality of portions of the 3D object. This may allowusing a reduced number of 2D images for representing the whole textureof the 3D object, thereby reducing the required disk space forrespective texture of respective 3D object.

According to a second aspect, there is provided a non-transitorycomputer-readable storage medium having stored thereon program codeportions for implementing the method according to the first aspect whenexecuted on a device having processing capabilities.

According to a third aspect, there is provided an electronic devicecomprising:

a display; and

-   -   circuitry configured to execute:        -   a 3D zooming function configured to determine a portion of a            3D object corresponding to a zoom level, and display the            determined portion of the 3D object including a texture            comprising a periodic pattern on the display; and        -   a 3D to 2D transition function configured to monitor the            zoom level and upon the zoom level reaching a zoom-in            threshold execute a 3D to 2D transition comprising:            -   identify a two-dimensional, 2D, image representing the                texture of the currently displayed portion of the 3D                object from a set of 2D images, each 2D image of the set                of 2D images representing the texture of the 3D object                at a specific of portion of the 3D object, wherein at                least one 2D image of the set of 2D images represents                the texture of the 3D object at a plurality of portions                of the 3D object; and            -   display the 2D image on the display, wherein a                resolution of the 2D image is higher than a resolution                of the texture of the portion of the 3D object at the                zoom-in threshold.

The above-mentioned features of the method, when applicable, apply tothe second and third aspects as well. In order to avoid unduerepetition, reference is made to the above.

The 3D to 2D transition may further comprise, while displaying thedetermined portion of the 3D object, shift a view direction of the 3Dobject to be parallel with a surface normal of the surface of the 3Dobject.

Generally, all terms used in the claims are to be interpreted accordingto their ordinary meaning in the technical field, unless explicitlydefined otherwise herein. All references to “a/an/the [element, device,component, means, step, etc.]” are to be interpreted openly as referringto at least one instance of said element, device, component, means,step, etc., unless explicitly stated otherwise. The steps of any methoddisclosed herein do not have to be performed in the exact orderdisclosed, unless explicitly stated.

BRIEF DESCRIPTION OF THE DRAWINGS

The above, as well as additional objects, features and advantages of thepresent invention, will be better understood through the followingillustrative and non-limiting detailed description of preferredembodiments, with reference to the appended drawings, where the samereference numerals will be used for similar elements, wherein:

FIG. 1 schematically shows a flowchart of an example of the disclosedmethod.

FIG. 2 schematically shows an aligning of a view angle while zooming inon a portion of a 3D object.

FIG. 3 schematically shows a 3D to 2D transition while zooming in on aportion of the 3D object.

FIG. 4 schematically shows an electronic device on which the method maybe implemented.

FIG. 5 shows a scene comprising a 3D object comprising a texture havinga periodic pattern.

DETAILED DESCRIPTION

The present invention will now be described more fully hereinafter withreference to the accompanying drawings, in which currently preferredembodiments of the invention are shown. This invention may, however, beembodied in many different forms and should not be construed as limitedto the embodiments set forth herein; rather, these embodiments areprovided for thoroughness and completeness, and to fully convey thescope of the invention to the skilled person.

In connection with FIG. 1, there is schematically shown a flowchart ofan example of a computer implemented method 100 for displaying detailsof a three-dimensional, 3D, object, 1000 wherein the texture comprises aperiodic pattern. The 3D object 1000 may be comprised in a 3D model,wherein the 3D model of the 3D object 1000 may be established using anyadequate 3D modelling technique, such as a 3D scanner, a 360-degreecamera, or the like. “The computer implemented method” 100 may bereferred to as “the method” 100 henceforth. FIGS. 2 and 3 maypictorially emphasize the described method steps below, hence preferablyto be inspected while reading the description of FIG. 1.

The method 100 comprises, while zooming in on the 3D object 1000,determining a portion 500 of the 3D object 1000, wherein the determinedportion 500 of the 3D object 1000 corresponds to a zoom level. Thewording “zooming in” typically, and also herein, refers to magnifying aportion of a digital representation, such as a surface of a 3D object1000, an ordinary 2D image, etc., for enhancing a rendering of imagedetails of such a portion. The zoom level may refer to a magnificationof a length element of a displayed element on a display. For instance,assuming a start position for viewing the displayed element has a zoomlevel of 100%, a zoom level of 200% implies that the length element ofthe displayed element appears twice as long on the display. Accordingly,an area portion of the displayed element appears four times larger insuch a situation.

The determined portion 500 of the 3D object 1000 may be a portion of thesurface of the 3D object 1000 that is significantly smaller than thetotal surface of the 3D object 1000. The determined portion 500 of the3D object 1000 may be a portion of the surface of the 3D object 1000that is significantly smaller than a total surface of the 3D object1000, seen from a specific view angle V when viewing the 3D object 1000.That is, from a specific view angle V of the 3D object 1000 and/or aspecific viewing distance from the 3D object 1000, it may be impossibleto see the total surface of the 3D object 1000. For instance, a “rearside” of a 3D object 1000 is invisible to a viewer when the 3D object1000 is viewed from the “front side”. Throughout this disclosure it isassumed that the 3D object 1000 is opaque. However, the method mayequally well be applied on an at least partly visually transparent 3Dobject 1000 such as a pleated thin curtain.

The 3D object 1000 may be a piece of furniture. By way of example, the3D object 1000 may be a sofa (see, e.g., FIG. 5), a table, a bookshelf,or the like. The 3D object 1000 may thus comprise an arbitrarily curvedsurface. By way of example, a sofa may be enclosed by a clothing, theclothing hence being a curved surface. However, the 3D object 1000 mayalso be a carpet comprising a surface being substantiallytwo-dimensional, 2D. The method applies equally well to such a 2Dsurface. For a 2D surface, the whole surface of interest may be viewedfrom a specific view angle and/or viewing distance, in contrast to,e.g., a sofa or a table.

The method 100 comprises, while zooming in on the 3D object 1000,displaying 20 the determined portion 500 of the 3D object 1000 includinga corresponding portion of the texture. The determined portion 500 maybe a surface portion of the 3D object 1000. The displaying 20 may bedone on any type of display 410 for displaying digital information. Byway of example, such a display 410 may be any one of a smartphonedisplay, a computer display, a tablet display, a TV, smart glasses, adedicated AR or VR device, or the like. The determined portion 500 maygenerally have a surface normal 510 pointing in a direction beingnon-parallel to a view direction on the determined portion. This isvisualized in, e.g., FIG. 2. That is, the view direction 270 may referto a vector pointing between an artificial substantially point-likelocation, being the artificial view location 250 of an artificial cameraor an eye of a viewer of the 3D object 1000, and a central point 260 onthe determined portion 500. The view direction 270 may be defined basedon a solid angle between the surface normal 510 of the determinedportion 500 and a vector 270 between the artificial view location 250and the central point 260 of the determined portion 500. The centralpoint 260 on the determined portion 500 may be substantially centrallylocated on the determined portion 500. For instance, the central point260 on the determined portion 500 may be a center of mass of thedetermined portion 500. The artificial view location 250 may be asimulated position of a camera being artificially located in the 3Dmodel. Hence, when viewing, e.g., a sofa in a 3D model from a certainartificial viewing point 250, a vast majority of all possible surfaceportions of the clothing of the sofa have surface normals pointing in adirection being non-parallel to the view direction. This is true alsowhen viewing a carpet from a viewing point straight above the carpet,provided the distance between the viewing point and the carpet is finiteand provided rendering of the carpet in any type of perspective view. Insuch a situation, a picture plane, i.e., the plane of the carpet, isparallel to two Cartesian axes of the scene, hence being a so-calledone-point perspective view, wherein parallel lines perpendicular to thepicture plane converge at a vanishing point. A similar situation may bea photo taken at a railway towards a corresponding vanishing point asignificant distance away, at an end of sight of the railway. At such aphoto, the rails are non-parallel. It is to be noted however, that ifsuch a carpet is rendered in an isometric view, the whole surface of thecarpet may be perpendicular to the view direction. The method 100 isapplicable to any type of view, such as any type of perspective view, anisometric view, etc. The view direction 270 may occasionally refer to aview angle. Hence, to enable viewing the determined portion 500, theangle V between the surface normal of the determined portion and theview direction may lie in the range of 0-90 degrees, provided thesurface of the determined portion is substantially flat. The wording“flat” generally, and also herein, refers to a surface being representedby a Euclidian plane spanned by two non-parallel basis vectors.

The method 100 comprises, upon the zoom level reaching a zoom-inthreshold, executing 30 a 3D to 2D transition. Preferably, a texturerecognition, further described below, of the texture of the determinedportion 500 has been performed when the zoom-in threshold has beenreached. Alternatively, the texture recognition is done whileapproaching the zoom-in threshold, the texture recognition therebystarting at an additional zoom-in threshold, the additional zoom-inthreshold corresponding to a smaller (more zoomed out) zoom level. Thetexture recognition may in such a situation be continuously performedwhile panning between the additional zoom-in threshold and the zoom-inthreshold. The 3D to 2D transition may utilize multithreading, i.e.,exploiting the ability of a CPU to provide a plurality of threads uponexecution.

The zoom-in threshold may be set by a user according a user preference.The zoom-in threshold may be predefined. The zoom-in threshold may beset such as depending on the context/use-case and/or object type. Thezoom-in threshold may represent a zoom-in to such a degree that aresolution of the displayed determined portion 500 falls below a minimalallowed resolution of the displayed determined portion. A resolutiontypically, and also herein, refers to a pixel density, e.g., the numberof pixels per length unit or the number of pixels per area unit. Theresolution may be measured in “pixels per inch” (ppi), i.e., the numberof pixels fitting on a line having a length of one inch (approximately2.54 cm). Hence a “high resolution” refers to a relatively high numberof pixels per inch, or, more generally, a relatively high number ofpixels per length unit. The minimal allowed resolution of the displayeddetermined portion may depend on a resolution of the display used todisplay the determined portion. Hence, a determined portion of the 3Dobject 1000 may be represented by a predefined number of pixels. Whenviewed from a certain viewing distance away from the 3D object 1000, andfrom a certain view direction 270, the resolution of the determinedportion 500 may allow a substantial zooming in on the determined portion500 while the determined portion 500 appear sharp on a display. That is,as long as the pixel density of the determined portion does not fallbelow the pixel density of the display while zooming in, zooming in maybe allowable. Zooming in further in such a situation may however bepossible to a certain extent. When referring to a “sharp” 3D or 2Dimage, the corresponding 3D or 2D image comprises digital informationcorresponding to a pixel density that is equal to or larger than aportion of the display on which the 3D or 2D image is displayed. Thatis, assuming an ordinary 2D image of raster-graphics type having aresolution of 1920×1080 pixels (HD resolution) is displayed on a displayhaving a resolution of 3840×2160 pixels (4 k resolution), the image maybe considered being sharp only if the image is zoomed in to such anextent that it fills a maximum of 25% of the display area in question.Zooming in further on such an image hence implies that a pixel of theimage is smoothen out to a plurality of pixels on the display, such thatthe image may appear unsharp/smooth/soft on the display. It is to benoted however that a level of detail of a 3D or 2D image is independentof the resolution of a display used to display the 3D or 2D image. It isfurther noted that many displays of today have apixel-density/display-size ratio so high that the human eye cannotdistinguish between individual pixels without magnifying means. An imagemay hence appear “sharp” to the human eye, despite technically beingunsharp according to the above. The method 100 is thereby not limited tothe resolution of a display used for the displaying described herein.

While continuing zooming in, after such a zoom-in threshold has beenreached, the 3D to 2D transition 30 of the digital representation isexecuted. In the 3D to 2D transition 30, a transition is executed fromdisplaying the determined portion of the 3D object 1000 to display a 2Dimage that corresponds to the determined portion 500 of the 3D object1000. The 2D image has a resolution that is higher than the resolutionof the determined portion 500 of the 3D object 1000. Hence, the user mayzoom in further, to a certain extent, on the 2D image without fallingbelow the resolution of the display, i.e., maintaining a sharp displayedrepresentation of the 2D image. The 2D image may be captured by anysuitable device. The 2D image may be captured by a digital camera. The2D image may be of any graphics type. The 2D image may be ofraster-graphics type. The 2D image may be of vector-graphics type. The2D image may be either or both of a raster-graphics type and avector-graphics type. Should the 2D image be of raster-graphics type,the 2D image may be converted to vector-graphics type by a built-inconverter. Similarly, an opposite such a conversion may be applied. Ifthe 2D image is of vector-graphics type the user may zoom in further,compared to a corresponding raster-graphics type of the 2D image, asvector graphics generally allows displaying a sharp image regardless ofthe zoom level. It is to be noted however, that 2D images ofvector-graphics type that have a high level of details, generallyrequires a relatively large amount of disk space. Hence, for detailedand/or large patterns of the texture of the surface of the 3D object1000, the 2D image may preferably be of raster-graphics type.Conversely, for simple and/or slowly varying patterns of the texture ofthe surface of the 3D object 1000, the 2D image may preferably be ofvector-graphics type. If the 2D image is of raster-graphics type theuser may zoom in to such an extent that the pixel density of the 2Dimage corresponds a pixel density of the display for displaying the 2Dimage. Zooming in even further may be possible, as is typically the casewhen zooming in images on an ordinary computer- or smartphone display,although the 2D image may then appear unsharp/soft.

The 3D to 2D transition comprises identifying a 2D image 40 representingthe texture of the currently displayed portion 500 of the 3D object 1000from a set of 2D images. Hence, the texture of the displayed portion 500of the 3D object 1000 may be displayed further refined by attributing a2D image having higher resolution compared to the correspondingrelatively low-resolution displayed determined portion 500 of the 3Dobject 1000. The identification of the 2D image may comprise finding abest correspondence between the portion of the 3D object 1000 and theimages of the set of 2D images. The finding of a best correspondencebetween the portion of the 3D object 1000 and the images of the set of2D images may comprise inputting a sample of the texture of a currentlydisplayed portion of the 3D object 1000 in a neural network beingtrained to output a 2D image representing a best match texture. Thesample of the texture of the currently displayed portion may be theentire texture in view or at least a portion of the texture in view.Alternatively, or additionally, the sample of the texture of thecurrently displayed portion may be a down-sampled version of thetexture. A down-sampled version of the texture may refer to the samplebeing represented by a smaller number of data points, such as a smallerpixel density, thereby corresponding to a smaller image resolution ofthe sample. Alternatively, or additionally, a down-sampled version ofthe texture may refer to the sample being represented by a smaller bitdepth, e.g., a smaller number of colors. Yet another option of adown-sampled version may be representation of the texture having areduced transparency range, e.g., by reducing/adjusting the number ofalpha channels, or reducing a complexity of possible light reflections,or the like. The down-sampled version of the texture may be obtainedutilizing any adequate compression technique.

Owing to the presumed periodic pattern of the surface of the 3D object1000, the total surface of the 3D object 1000 may be represented by anumber of 2D images corresponding to area being substantially smallerthan the area of the total surface of the 3D object 1000. Hence, atleast one 2D image may be used at a plurality of locations of the 3Dobject 1000 due to the periodic pattern of the texture of the 3D object1000. Conversely, each 2D image of the set of 2D images may representthe texture of the 3D object 1000 at a plurality of portions of the 3Dobject 1000. Any type of neural network scheme is possible to use fortraining to output a 2D image representing a best match texture. Themethod 100 may thus comprise learning to recognize a displayed portionof the 3D object 1000 and subsequently choose a 2D image correspondingto the displayed portion of the 3D object 1000 to further enhancedetails in the texture of the displayed portion.

Alternatively, the finding of a best correspondence between the portionof the 3D object 1000 and the images of the set of 2D images maycomprise any ordinary image comparison technique. Such an imagecomparison technique may be an algorithm for comparing images, patternsor textures of an object to images of a database that are similar to theobject of the image and use the outcome for determining, e.g., aperiodically occurring pattern. The image comparison algorithm mayinclude one or more of a pixel-to-pixel comparison, edgedetection/comparison, keypoint matching, histogram methods, keypointrecognition using randomized trees, or the like. The matching of the 2Dimage towards the surface of the 3D object 1000 may be donesubstantially in real-time by comparing a current 3D view with theimages of the set of 2D images such that the most appropriate 2D imageis selected.

The finding of a best correspondence between the portion of the 3Dobject 1000 and the images of the set of 2D images may compriseaccessing a conversion table between specific portions of the 3D object1000 and the 2D images in the set of 2D images. Such a conversion tablemay comprise a table between the surface of the 3D object 1000 and theimages of the set of 2D images, wherein positions, sizes and/or relevantrotations of the images may be determined. The conversion table and the2D images may be transferred to the user's electronic device and storedthereon while the user chooses to explore a specific 3D model.

Should any or both of the above-mentioned neural-network technique andthe ordinary image comparison technique be too slow or demand too muchdata to be transferred, the conversion table may alternatively be used,wherein the conversion table may comprise precomputed positions, sizesand/or relevant rotations of the images of the set of 2D images to bestored on a server, a device, or the like.

The method 100 may recognize a displayed portion 500 despite asignificant angle V between the surface normal 510 of the surface of thedisplayed portion 500 and the view direction 270 when displaying thedisplayed portion 500. Such a recognition may utilize any adequatelinear transformation algorithm, such as a direct linear transformationscheme, to minimize errors and/or enhancing the learning rate whilerecognizing the displayed portion 500.

The recognition may comprise utilizing any adequate manifoldtransformation for transforming a possibly curved surface of thedisplayed portion to a corresponding flat surface to be compared to acorresponding 2D image of the set of 2D images, to minimize errorsand/or enhancing the learning rate while recognizing the displayedportion 500.

Each such a 2D image of the set of 2D images represents the texture ofthe 3D object 1000 at a specific portion of the 3D object 1000. At leastone 2D image of the set of 2D images represents the texture of the 3Dobject 1000 at a plurality of portions of the 3D object 1000. Assumingtemporarily that the surface of the 3D object 1000 is flat, the periodicpattern of the surface of the 3D object 1000 is represented by a numberof “unit cells”, wherein each unit cell is essentially identical to anyother unit cell of the surface, thereby repeating itself over thesurface to form a set of densely packed tiles to represent the wholesurface of the 3D object 1000. Hence, the end points of any translationvector of the periodic pattern of the surface contain similar digitalinformation, apart from possible light effects, shadowing, or the like.A user zooming in on an arbitrarily chosen specific portion of the 3Dobject 1000 hence likely ends up by viewing a plurality of adjacenttiles. The method 100 may in such a situation stitch together such aplurality of tiles to seamlessly represent the specific portion of the3D surface by a corresponding 2D image. This applies equally well tocurved surfaces as of the above-mentioned teachings. The number of 2Dimages representing the whole surface of the 3D object 1000 may be lessthan 20, or more preferably less than ten.

The surface of the 3D object 1000 may comprise a plurality of materials,wherein respective material of the plurality of materials comprises acertain periodic pattern. The method and the electronic device describedabove and below, may in such a situation be applied to respectivematerial of the plurality of materials. For instance, the number of 2Dimages representing respective material of a plurality of materials ofthe surface of the 3D object 1000 may be less than 20, or morepreferably less than 10. Optionally, the execution of a 3D to 2Dtransition 30 may further comprise, while displaying the determinedportion of the 3D object 1000, shifting 50 a view direction of the 3Dobject 1000 to be parallel with a surface normal of the surface of thedetermined portion of the 3D object 1000. If the determined portion ofthe 3D object is curved, an average surface normal can be calculated tobe used for the shifting 50 of the view direction. The shifting 50 ofthe view angle is schematically displayed in FIG. 2. The wording“parallel” may herein refer to substantially parallel. Hence, an anglebetween two “parallel” lines as referred to herein may lie in the range0°±5°. Further, the wording “parallel” does herein excludingdistinguishing between parallel and antiparallellines/vectors/directions, etc. For instance, when viewing the displayedportion straight from above, a surface normal vector of the displayedportion may be opposingly directed towards the view direction, thesurface normal and the view direction thereby, being mathematicallycorrect, being “antiparallel”. A similar tolerance as that described inconnection to the wording “parallel” above may apply for the wording“perpendicular”. That is, the angle between two “perpendicular” lines asreferred to herein may lie in the range 90°±5°. A user may hence startviewing the surface of the 3D object 1000 at a start position 200 andchoose a region of the 3D object 1000 of interest, the regioncorresponding to the determined portion. Again, the determined portionmay presumably be flat, such that each point on the determined portionis defined by two space coordinates. The view angle V may be defined asa solid angle V between the surface normal 510 of the determined portion500 and an artificial viewing point 250 in the 3D model. The shifting 50of the view angle V may continuously converging towards being parallelto the surface normal 510 of the surface of the 3D object 1000.. Theshifting 50 of the view angle V may be linearly proportional to the zoomlevel. The shifting 50 of the view angle V may be linearly proportionalto the zoom-in duration, i.e., the view angle V may shift X degrees perunit of time during a zoom-in session. The shifting 50 of the view angleV may be subject to a non-linear relationship with the zoom level. Forinstance, the rate of change of the view angle V may be higher in abeginning of a zoom-in session compared to an end of the zoom-insession. An opposing situation may be equally applicable to the method,i.e., the rate of change of the view angle V may be lower in a beginningof a zoom-in session compared to an end of the zoom-in session.Alternatively, the view angle V may be substantially constant during abeginning of a zoom-in session until a certain zoom level is reached,subsequently start shifting the view angle as of the above, or viceversa. The shifting of the view angle may be performed during a timeperiod of the order of seconds.

In other embodiments, the shifting 50 of view angle is not performed.For instance, a present view may be faded between a 3D view of the 3Dobject 1000 and the 2D image when the view angle is non-parallel to asurface normal of the determined portion. Another example is that apresent view may momentarily shift between a 3D view of the determinedportion and a corresponding 2D image.

The method 100 comprises displaying 60 the 2D image, wherein aresolution of the 2D image is higher than a resolution of the texture ofthe portion of the 3D object 1000 at the zoom-in threshold. In the caseof performing the step 50 of shifting, the step of displaying 60 the 2Dimage may be made upon the shifting 50 is completed.

Upon executing 30 the 3D to 2D transition, the 3D to 2D transition maycomprise displaying a merge of the portion of the 3D object 1000 and the2D image. Such a merge of the portion of the 3D object 1000 and the 2Dimage may be set to gradually change from 100% of the portion of the 3Dobject 1000 to 100% of the 2D image. Such a gradual change may be doneduring a merge duration being a period of time of the order of seconds.Such a gradual change may be a linear gradual change in that the mergelinearly changes from 100% of the portion of the 3D object 1000 to 100%of the 2D image during the merge duration. The gradual change mayhowever exhibit any non-linear gradual change, e.g., the mergequadratically changing from 100% of the portion of the 3D object 1000 to100% of the 2D image during the duration of the merge.

The method 100 may further comprise determining an identifier of the 3Dobject 1000, thereby determining the set of 2D images. The identifiermay correspond to a class of objects, e.g., sofas, pillows, carpets, orthe like. The identifier may also correspond to a specific type ofobject within a class of objects, such as a specific type of sofa or aspecific type of pillow. The identifier may identify the pattern of thetexture. It should be noted that a 3D object 1000 may comprise aplurality of patterns (e.g. different materials). In this case, the stepof determining the identifier may be based on the determined portion ofthe 3D object 1000 (i.e. a current field of view of the 3D object 1000),wherein different portions of the 3D object 1000 are tagged or similarwith different identifiers. The identifier(s) may be included asmetadata of the 3D object 1000. The determining of an identifier maycomprise a neural network, an image search algorithm, or the like.

The method may further comprise zooming and/or panning in the 2D image.The displaying of the 2D image, i.e., after the steps 10-40 and,optionally, step 50, have been done, may occasionally be referred to as2D mode. Similarly, the displaying of the 3D object 1000 or thedetermined portion 500 of the 3D object 1000, i.e., before the 3D to 2Dtransition, may refer to being in 3D mode. As with normal 2D imagesdisplayed on normal displays, zooming in may be possible when viewingthe 2D image. The resolution of the 2D image may be higher than theresolution of the display for displaying the 2D image. Hence, zooming inon the 2D image from a full-display mode of the 2D image, i.e., when the2D image is substantially fitted to the full area of the display, mayallow representation of a sharp zoomed in portion of the 2D image. Thismay be applicable when, e.g., when a 2D image having a 4k resolution isdisplayed on an HD display. While in 2D mode, the user may optionallychange view angle, view perspective, rotation, shading, light effects,and/or the like, of the 2D image to enhance the realism of the textureof the surface of the 3D object 1000. Using haptics, the user may alsohave an experience of touch by applying forces, vibrations, or motionsto the haptics equipment worn by the user, depending on the texture andthe zoom level.

Normal panning may be possible in any zoom level of the 2D image, eitherin a normal view of the 2D image, i.e., the view direction beingperpendicular to the display, or after changing view perspective,rotation, or the like. Panning in any zoom level of the 2D image mayfurther be possible along the whole surface of the 3D object 1000, as ofthe presumed periodic pattern of the texture. In such a situation, themethod 100 may account for possible curved surface portions of the 3Dobject 1000, light effects, or the like.

The method 100 may further comprise, upon a zoom level in the 2D imagereaches a zoom-out threshold, executing a 2D to 3D transition comprisingdisplaying a portion of the 3D object 1000 corresponding to a currentlydisplayed portion of the 2D image. The zoom-out threshold may be a zoomlevel such that being similar to the zoom-in threshold reached byzooming in on the 3D object 1000. Hence, the method 100 may, at leastpartly, comprise a reverse ordering of the steps 10-40, the optionalstep 50, and step 60, allowing zooming out from a zoomed in 2D image todisplaying a corresponding portion of the surface of the 3D object 1000.Zooming out to display the 3D object 1000 in its entirety may further bepossible. A trace of a zoom-out path, while zooming out from a centralpoint 260 of the determined portion 500, may be similar to a trace of apreceding zoom-in path on the determined portion 500. That is, the viewdirection may, with respect to the the surface normal 510 of thedetermined portion 500, be similar to a corresponding zoom-in path foreach viewing distance. Alternatively, the zoom-out path may be differentfrom the zoom-in path. For instance, a zoom-out path may be such thatthe view direction is parallel to the surface normal of the determinedportion during the entire zoom-out path. Provided panning has been done,while in a zoomed in position of the 2D image, the zoom-out path mayoriginate from a centered point of a presently displayed content andproceed along a path being parallel, i.e., shifted by a previouslypanned distance, to the zoom-in path. Alternatively, the zoom-out pathgradually converges to the zoom-in path such that the start position maybe reached when regaining the viewing distance of the start position.

Turning to FIG. 3, there is shown, highly schematically, a subset of thesteps of the method 100. FIG. 3 aims to further facilitate theunderstanding of the method 100. In FIG. 3, the view direction isdirected perpendicularly into the display/paper, in contrast to FIG. 2,in which the view direction is seen from a side view. More specifically,a determined portion 500 comprising a periodically occurring pattern 520is shown with different view directions 310;320;330 with respect to asurface normal of the determined portion 500. At 310 the angle between anormal vector of the surface of the determined portion and the viewdirection is relatively large. While displaying the determined portion500 of the 3D object 1000, upon zooming in on the determined portion500, the angle between the normal vector and the view directiongradually or stepwise decreases. Zooming in further (iii), still whiledisplaying the determined portion 500 of the 3D object 1000, theabove-mentioned angle eventually approaches zero degrees, such that thenormal vector 510 of the surface of the determined portion 500 and theview direction becomes parallel. For convenience, only the apparentshift in view direction, and not the apparent magnification due to theincreasing zoom level, is shown in 310, 320 and 330. Further, theperiodically occurring pattern of the texture of the determined portion500 is here depicted with an excessively scarce image resolution toemphasize the difference in resolution of viewing the determined portion500 in 3D mode 310;320;330 compared to viewing the determined portion500 in 2D mode 340;350. After the above-mentioned 3D to 2D transitionhas been executed, a 2D image 340 of the determined portion 500 isdisplayed, the 2D image having a higher resolution than the resolutionof the texture of the determined portion 500 of the 3D object 1000 atthe zoom-in threshold. Being in the 2D mode enables further zooming in350 for exploring details of the texture of the determined portion 500.

In connection with FIG. 4 an electronic device 400 on which thedisclosed method 100 is implemented.

The electronic device 400 comprises a display 410. The display 410 maybe any type of display for displaying digital information. By way ofexample, the display may be any one of a smartphone display, a computerdisplay, a tablet display, a TV, smart glasses, a dedicated AR or VRdevice, or the like. The display 410 may be an integral part of theelectronic device 400. The display 410 may be freestanding such thatbeing configured to wirelessly communicate with circuitry 420 of theelectronic device 400, e.g., by Bluetooth, Wi-Fi, or the like, thecircuitry 420 being further described below. The display 410 may alsocommunicate with the circuitry 420 by wire.

The electronic device 400 comprises circuitry 420.

The circuitry 420 is configured to carry out overall control offunctions and operations of the electronic device 400. The circuitry 420may include a processor, such as a central processing unit (CPU),microcontroller, or microprocessor. The processor is configured toexecute program code stored in the circuitry to carry out functions andoperations of the electronic device 400.

Executable functions, further described below, may be stored on amemory. The memory may be one or more of a buffer, a flash memory, ahard drive, a removable media, a volatile memory, a non-volatile memory,a random access memory (RAM), or other suitable devices. In a typicalarrangement, the memory may include a non-volatile memory for long termdata storage and a volatile memory that functions as system memory forthe circuitry 420. The memory may exchange data with the circuitry 420over a data bus. Accompanying control lines and an address bus betweenthe memory and the circuitry 420 may be present.

Functions and operations of the circuitry 420 may be embodied in theform of executable logic routines, e.g., computer-code portions,software programs, etc., that are stored on a non-transitory computerreadable medium, e.g., the memory, of the electronic device 400 and areexecuted by the circuitry 420 by, e.g., using the processor. Thefunctions and operations of the electronic device 400 may be astand-alone software application or form a part of a softwareapplication that carries out additional tasks related to the electronicdevice 400. The described functions and operations may be considering amethod that the corresponding device is configured to carry out. Also,while the described functions and operations may be implemented in asoftware, such functionality may as well be carried out via dedicatedhardware or firmware, or some combination of hardware, firmware and/orsoftware.

The circuitry 420 is configured to execute a 3D zooming function 422.The 3D zooming function 422 is configured to determine a portion 500 ofa 3D object 1000 corresponding to a zoom level and display thedetermined portion 500 of the 3D object 1000 including a texturecomprising a periodic pattern 520 on the display 410.

The circuitry 420 is configured to execute a 3D to 2D transitionfunction 424 configured to monitor the zoom level upon the zoom levelreaching a zoom-in threshold execute a 3D to 2D transition comprisingidentify a 2D image representing the texture of the currently displayedportion of the 3D object 1000 from a set of 2D images, each 2D image ofthe set of 2D images representing the texture of the 3D object 1000 at aspecific of portion of the 3D object 1000, wherein at least one 2D imageof the set of 2D images represents the texture of the 3D object 1000 ata plurality of portions of the 3D object 1000, and display the 2D imageon the display 410, wherein a resolution of the 2D image is higher thana resolution of the texture of the portion 500 of the 3D object 1000 atthe zoom-in threshold.

In connection with FIG. 5, there is shown an at least partly virtualscene comprising a 3D object 1000 comprising a texture having a periodicpattern. The scene may be fully computer generated scene, or a partlycomputer generated scene such as an augmented reality scene. The 3Dobject 1000 is here exemplified as a sofa comprising periodicallyoccurring stars as a pattern of a clothing. The user may zoom in on adetermined portion 500 of the texture. Here, the determined portion 500has a surface normal, or, more correctly, a plurality of slightlydifferently directed surface normals, being non-parallel to a viewdirection. In FIG. 5, the view direction is perpendicular to a displayfor viewing FIG. 5, or a paper copy of FIG. 5, provided FIG. 5 isprinted, etc. At a first display mode 510, the determined portion 500 iszoomed in to a certain extent, wherein an angle between a surface normalof the determined portion 500 and the view direction is relativelylarge. At a second display mode 520, a shift of the view direction ofthe determined portion has optionally been done to be parallel with asurface normal of the determined portion 500. At a third display mode530, the 3D to 2D transition has been executed for displaying acorresponding 2D image of the determined portion 500, wherein theresolution of the 2D image is higher than the resolution of the textureof the determined portion of the 3D object at the zoom-in threshold. Theperiodically occurring pattern of the texture of the determined portion500 is here depicted excessively scarce to emphasize the difference inresolution of viewing the determined portion 500 in the first and thesecond display mode 510;520 compared to viewing the determined portion500 in the third display mode 530. Details associated with the displaymodes 510;520;530 are described in the text above.

Other features and embodiments of the electronic device may beapplicable to the above-mentioned specification of the method 100.

The person skilled in the art realizes that the present invention by nomeans is limited to the preferred embodiments described above. On thecontrary, many modifications and variations are possible within thescope of the appended claims.

Additionally, variations to the disclosed embodiments can be understoodand effected by the skilled person in practicing the claimed invention,from a study of the drawings, the disclosure, and the appended claims.

What is claimed is:
 1. A computer implemented method for displayingdetails of a texture of a three-dimensional, 3D, object, the texturecomprising a periodic pattern, the method comprising: while zooming inon the 3D object: determining a portion of the 3D object, wherein thedetermined portion of the 3D object corresponds to a zoom level; anddisplaying the determined portion of the 3D object including acorresponding portion of the texture; and upon the zoom level reaching azoom-in threshold, executing a 3D to 2D transition comprising:identifying a two-dimensional, 2D, image representing the texture of thecurrently displayed portion of the 3D object from a set of 2D images,each 2D image of the set of 2D images representing the texture of the 3Dobject at a specific of portion of the 3D object, wherein at least one2D image of the set of 2D images represents the texture of the 3D objectat a plurality of portions of the 3D object; and displaying the 2Dimage, wherein a resolution of the 2D image is higher than a resolutionof the texture of the portion of the 3D object at the zoom-in threshold.2. The method according to claim 1, wherein the 3D to 2D transitionfurther comprises, while displaying the determined portion of the 3Dobject, shifting a view direction of the 3D object to be parallel with asurface normal of the surface of the 3D object,
 3. The method accordingto claim 1, further comprising determining an identifier of the 3Dobject, thereby determining the set of 2D images.
 4. The methodaccording to claim 1, wherein identifying the 2D image comprises findinga best correspondence between the portion of the 3D object and theimages of the set of 2D images.
 5. The method according to claim 2,wherein finding a best correspondence between the portion of the 3Dobject and the images of the set of 2D images comprises inputting asample of the texture of a currently displayed portion of the 3D objectin a neural network being trained to output a 2D image representing abest match texture.
 6. The method according to claim 3, wherein findinga best correspondence between the portion of the 3D object and theimages of the set of 2D images comprises an image comparison between asample of the texture of a currently displayed portion of the 3D objectand the textures represented by the 2D images in the set of 2D images.7. The method according to claim 3, wherein finding a bestcorrespondence between the portion of the 3D object and the images ofthe set of 2D images comprises accessing a conversion table betweenspecific portions of the 3D object and the 2D images in the set of 2Dimages.
 8. The method according to claim 1, wherein the 3D to 2Dtransition further comprises displaying a merge of the portion of the 3Dobject and the 2D image.
 9. The method according to claim 6, wherein themerge of the portion of the 3D object and the 2D image is set togradually change from 100% of the portion of the 3D object to 100% ofthe 2D image.
 10. The method according to claim 1, further comprisingzooming and/or panning in the 2D image.
 11. The method according toclaim 8, further comprising, upon a zoom level in the 2D image reaches azoom-out threshold, executing a 2D to 3D transition comprisingdisplaying a portion of the 3D object corresponding to a currentlydisplayed portion of the 2D image.
 12. The method according to claim 1,wherein each 2D image of the set of 2D images representing the textureof the 3D object at a plurality of portions of the 3D object.
 13. Anon-transitory computer-readable storage medium having stored thereonprogram code portions for implementing the method according to claim 1when executed on a device having processing capabilities.
 14. Anelectronic device comprising: a display; and circuitry configured toexecute: a 3D zooming function configured to determine a portion of a 3Dobject corresponding to a zoom level, and display the determined portionof the 3D object including a texture comprising a periodic pattern onthe display; and a 3D to 2D transition function configured to monitorthe zoom level and upon the zoom level reaching a zoom-in thresholdexecute a 3D to 2D transition comprising: identify a two-dimensional,2D, image representing the texture of the currently displayed portion ofthe 3D object from a set of 2D images, each 2D image of the set of 2Dimages representing the texture of the 3D object at a specific ofportion of the 3D object, wherein at least one 2D image of the set of 2Dimages represents the texture of the 3D object at a plurality ofportions of the 3D object; and display the 2D image on the display,wherein a resolution of the 2D image is higher than a resolution of thetexture of the portion of the 3D object at the zoom-in threshold. 15.The electronic device according to claim 14, wherein the 3D to 2Dtransition further comprises, while displaying the determined portion ofthe 3D object, shift a view direction of the 3D object to be parallelwith a surface normal of the surface of the 3D object.